Attention and self-attention in random forests

New models of random forests jointly using the attention and self-attention mechanisms are proposed for solving regression problem. The can be regarded as extensions attention-based forest whose idea stems from applying a combination Nadaraya–Watson kernel Huber’s contamination model to forests. aims capture dependencies tree predictions remove noise or anomalous in forest. module is trained with computing weights. It shown that training process weights reduced single quadratic linear optimization Three modifications compared. A specific multi-head also considered. Heads obtained by changing its tuning parameters including parameter models. combinations verified compared other several datasets. code implementing corresponding algorithms publicly available.